Crowdsourcing the identification of studies for COVID-19-related Cochrane Rapid Reviews.

BACKGROUND: Utilisation of crowdsourcing within evidence synthesis has increased over the last decade. Crowdsourcing platform Cochrane Crowd has engaged a global community of 22,000 people from 170 countries. The COVID-19 pandemic presented an opportunity to engage the community and keep up with the exponential output of COVID-19 research. AIMS: To test whether a crowd could accurately assess study eligibility for reviews under time constraints. OUTCOME MEASURES: time taken to complete each task, time to produce required training modules, crowd sensitivity, specificity and crowd consensus. METHODS: We created four crowd tasks, corresponding to four Cochrane COVID-19 Rapid Reviews. The search results of each were uploaded and an interactive training module was developed for each task. Contributors who had participated in another COVID-19 task were invited to participate. Each task was live for 48-h. The final inclusion and exclusion decisions made by the core author team were used as the reference standard. RESULTS: Across all four reviews 14,299 records were screened by 101 crowd contributors. The crowd completed each screening task within 48-h for three reviews and in 52 h for one. Sensitivity ranged from 94% to 100%. Four studies, out of a total of 109, were incorrectly rejected by the crowd. However, their absence ultimately would not have altered the conclusions of the reviews. Crowd consensus ranged from 71% to 92% across the four reviews. CONCLUSION: Crowdsourcing can play a valuable role in study identification and offers willing contributors the opportunity to help identify COVID-19 research for rapid evidence syntheses.

Crowdsourcing citation-screening in a mixed-studies systematic review: a feasibility study.

BACKGROUND: Crowdsourcing engages the help of large numbers of people in tasks, activities or projects, usually via the internet. One application of crowdsourcing is the screening of citations for inclusion in a systematic review. There is evidence that a 'Crowd' of non-specialists can reliably identify quantitative studies, such as randomized controlled trials, through the assessment of study titles and abstracts. In this feasibility study, we investigated crowd performance of an online, topic-based citation-screening task, assessing titles and abstracts for inclusion in a single mixed-studies systematic review. METHODS: This study was embedded within a mixed studies systematic review of maternity care, exploring the effects of training healthcare professionals in intrapartum cardiotocography. Citation-screening was undertaken via Cochrane Crowd, an online citizen science platform enabling volunteers to contribute to a range of tasks identifying evidence in health and healthcare. Contributors were recruited from users registered with Cochrane Crowd. Following completion of task-specific online training, the crowd and the review team independently screened 9546 titles and abstracts. The screening task was subsequently repeated with a new crowd following minor changes to the crowd agreement algorithm based on findings from the first screening task. We assessed the crowd decisions against the review team categorizations (the 'gold standard'), measuring sensitivity, specificity, time and task engagement. RESULTS: Seventy-eight crowd contributors completed the first screening task. Sensitivity (the crowd's ability to correctly identify studies included within the review) was 84% (N = 42/50), and specificity (the crowd's ability to correctly identify excluded studies) was 99% (N = 9373/9493). Task completion was 33 h for the crowd and 410 h for the review team; mean time to classify each record was 6.06 s for each crowd participant and 3.96 s for review team members. Replicating this task with 85 new contributors and an altered agreement algorithm found 94% sensitivity (N = 48/50) and 98% specificity (N = 9348/9493). Contributors reported positive experiences of the task. CONCLUSION: It might be feasible to recruit and train a crowd to accurately perform topic-based citation-screening for mixed studies systematic reviews, though resource expended on the necessary customised training required should be factored in. In the face of long review production times, crowd screening may enable a more time-efficient conduct of reviews, with minimal reduction of citation-screening accuracy, but further research is needed.

An evaluation of Cochrane Crowd found that crowdsourcing produced accurate results in identifying randomized trials.

BACKGROUND AND OBJECTIVES: Filtering the deluge of new research to facilitate evidence synthesis has proven to be unmanageable using current paradigms of search and retrieval. Crowdsourcing, a way of harnessing the collective effort of a "crowd" of people, has the potential to support evidence synthesis by addressing this information overload created by the exponential growth in primary research outputs. Cochrane Crowd, Cochrane's citizen science platform, offers a range of tasks aimed at identifying studies related to health care. Accompanying each task are brief, interactive training modules, and agreement algorithms that help ensure accurate collective decision-making.The aims of the study were to evaluate the performance of Cochrane Crowd in terms of its accuracy, capacity, and autonomy and to examine contributor engagement across three tasks aimed at identifying randomized trials. STUDY DESIGN AND SETTING: Crowd accuracy was evaluated by measuring the sensitivity and specificity of crowd screening decisions on a sample of titles and abstracts, compared with "quasi gold-standard" decisions about the same records using the conventional methods of dual screening. Crowd capacity, in the form of output volume, was evaluated by measuring the number of records processed by the crowd, compared with baseline. Crowd autonomy, the capability of the crowd to produce accurate collectively derived decisions without the need for expert resolution, was measured by the proportion of records that needed resolving by an expert. RESULTS: The Cochrane Crowd community currently has 18,897 contributors from 163 countries. Collectively, the Crowd has processed 1,021,227 records, helping to identify 178,437 reports of randomized controlled trials (RCTs) for Cochrane's Central Register of Controlled Trials. The sensitivity for each task was 99.1% for the RCT identification task (RCT ID), 99.7% for the RCT identification task of trials from ClinicalTrials.gov (CT ID), and 97.7% for the identification of RCTs from the International Clinical Trials Registry Platform (ICTRP ID). The specificity for each task was 99% for RCT ID, 98.6% for CT ID, and 99.1% for CT ICTRP ID. The capacity of the combined Crowd and machine learning workflow has increased fivefold in 6 years, compared with baseline. The proportion of records requiring expert resolution across the tasks ranged from 16.6% to 19.7%. CONCLUSION: Cochrane Crowd is sufficiently accurate and scalable to keep pace with the current rate of publication (and registration) of new primary studies. It has also proved to be a popular, efficient, and accurate way for a large number of people to play an important voluntary role in health evidence production. Cochrane Crowd is now an established part of Cochrane's effort to manage the deluge of primary research being produced.

Citation screening using crowdsourcing and machine learning produced accurate results: Evaluation of Cochrane's modified Screen4Me service.

OBJECTIVES: To assess the feasibility of a modified workflow that uses machine learning and crowdsourcing to identify studies for potential inclusion in a systematic review. STUDY DESIGN AND SETTING: This was a substudy to a larger randomized study; the main study sought to assess the performance of single screening search results versus dual screening. This substudy assessed the performance in identifying relevant randomized controlled trials (RCTs) for a published Cochrane review of a modified version of Cochrane's Screen4Me workflow which uses crowdsourcing and machine learning. We included participants who had signed up for the main study but who were not eligible to be randomized to the two main arms of that study. The records were put through the modified workflow where a machine learning classifier divided the data set into "Not RCTs" and "Possible RCTs." The records deemed "Possible RCTs" were then loaded into a task created on the Cochrane Crowd platform, and participants classified those records as either "Potentially relevant" or "Not relevant" to the review. Using a prespecified agreement algorithm, we calculated the performance of the crowd in correctly identifying the studies that were included in the review (sensitivity) and correctly rejecting those that were not included (specificity). RESULTS: The RCT machine learning classifier did not reject any of the included studies. In terms of the crowd, 112 participants were included in this substudy. Of these, 81 completed the training module and went on to screen records in the live task. Applying the Cochrane Crowd agreement algorithm, the crowd achieved 100% sensitivity and 80.71% specificity. CONCLUSIONS: Using a crowd to screen search results for systematic reviews can be an accurate method as long as the agreement algorithm in place is robust. TRIAL REGISTRATION: Open Science Framework: https://osf.io/3jyqt.

Which are the most sensitive search filters to identify randomized controlled trials in MEDLINE?

OBJECTIVE: The Cochrane Handbook of Systematic Reviews contains search filters to find randomized controlled trials (RCTs) in Ovid MEDLINE: one maximizing sensitivity and another balancing sensitivity and precision. These filters were originally published in 1994 and were adapted and updated in 2008. To determine the performance of these filters, the authors tested them and thirty-six other MEDLINE filters against a large new gold standard set of relevant records. METHODS: We identified a gold standard set of RCT reports published in 2016 from the Cochrane CENTRAL database of controlled clinical trials. We retrieved the records in Ovid MEDLINE and combined these with each RCT filter. We calculated their sensitivity, relative precision, and f-scores. RESULTS: The gold standard comprised 27,617 records. MEDLINE searches were run on July 16, 2019. The most sensitive RCT filter was Duggan et al. (sensitivity=0.99). The Cochrane sensitivity-maximizing RCT filter had a sensitivity of 0.96 but was more precise than Duggan et al. (0.14 compared to 0.04 for Duggan). The most precise RCT filters had 0.97 relative precision and 0.83 sensitivity. CONCLUSIONS: The Cochrane Ovid MEDLINE sensitivity-maximizing RCT filter can continue to be used by Cochrane reviewers and to populate CENTRAL, as it has very high sensitivity and a slightly better precision relative to more sensitive filters. The results of this study, which used a very large gold standard to compare the performance of all known RCT filters, allows searchers to make better informed decisions about which filters to use for their work.

Single-reviewer abstract screening missed 13 percent of relevant studies: a crowd-based, randomized controlled trial.

OBJECTIVES: To determine the accuracy of single-reviewer screening in correctly classifying abstracts as relevant or irrelevant for literature reviews. STUDY DESIGN AND SETTING: We conducted a crowd-based, parallel-group randomized controlled trial. Using the Cochrane Crowd platform, we randomly assigned eligible participants to 100 abstracts each of a pharmacological or a public health topic. After completing a training exercise, participants screened abstracts online based on predefined inclusion and exclusion criteria. We calculated sensitivities and specificities of single- and dual-reviewer screening using two published systematic reviews as reference standards. RESULTS: Two hundred and eighty participants made 24,942 screening decisions on 2,000 randomly selected abstracts from the reference standard reviews. On average, each abstract was screened 12 times. Overall, single-reviewer abstract screening missed 13% of relevant studies (sensitivity: 86.6%; 95% confidence interval [CI], 80.6%-91.2%). By comparison, dual-reviewer abstract screening missed 3% of relevant studies (sensitivity: 97.5%; 95% CI, 95.1%-98.8%). The corresponding specificities were 79.2% (95% CI, 77.4%-80.9%) and 68.7% (95% CI, 66.4%-71.0%), respectively. CONCLUSIONS: Single-reviewer abstract screening does not appear to fulfill the high methodological standards that decisionmakers expect from systematic reviews. It may be a viable option for rapid reviews, which deliberately lower methodological standards to provide decision makers with accelerated evidence synthesis products.

Translating the Cochrane EMBASE RCT filter from the Ovid interface to Embase.com: a case study.

BACKGROUND: Information specialists frequently translate search filters from one interface to another. Publications advise that translation can be complex and should be undertaken carefully. OBJECTIVES: To investigate the issues arising when translating the Cochrane Embase RCT search filter from one interface (Ovid) to another (Embase.com). METHODS: We drafted a translation of the Cochrane Ovid RCT filter to run in Embase.com. We compared the line-by-line results of the Ovid filter with the results of the translation. We revised the filter. We identified differences between database versions including records with different publication years and subject headings. Some records were in Embase in one interface but not in the other. We encountered expected interface differences relating to proximity operators. We also encountered unexpected interface issues around truncation and the use of the original title or original abstract field. DISCUSSION: Filter conversion is challenging and time consuming revealing unexpected differences in interfaces and databases. Careful planning can pre-empt some issues, but others may only emerge during testing. We identified interface anomalies that have led database publishers to review aspects of the way their interfaces work. CONCLUSIONS: Translators should be vigilant for known and unknown differences in both interfaces and database versions.

Development of a search filter to identify reports of controlled clinical trials within CINAHL Plus.

BACKGROUND: Evidence synthesis reviews in health care rely on the efficient identification of research evidence, particularly evidence from randomised controlled trials (RCTs). There are no recently validated filters to identify RCTs in the Cumulative Index to Nursing and Allied Health Literature (CINAHL Plus). OBJECTIVES: To develop, test and validate a search filter to identify reports of RCTs from CINAHL Plus. METHODS: Nine sets of relevant and irrelevant records were identified to develop and test search filters iteratively. Two sets were used to validate the sensitivity and precision of the filters. The performance of two previously published filters and the filter built into EBSCOhost was evaluated. RESULTS: We present a validated filter which offers sensitivity of 0.88 (95% CI: 0.77-0.95) and precision of 0.36 (95% CI: 0.31-0.41). This is comparable to the sensitivity of published filters, but has much better precision. CONCLUSIONS: A sensitive and precise filter, developed using records selected based on title and abstract information, is available for identifying reports of RCTs in the CINAHL Plus database via EBSCOhost. Using this filter is likely to reduce the number of results needing to be screened to a quarter of those retrieved by other published filters.

Topical treatments for chronic plaque psoriasis: an abridged Cochrane systematic review.

BACKGROUND: Chronic plaque psoriasis is the most common type of psoriasis and is characterized by redness, thickness, and scaling. First-line management is with topical treatments. OBJECTIVE: We sought to undertake a Cochrane review of topical treatments for chronic plaque psoriasis. METHODS: We systematically searched major databases for randomized controlled trials. Trials reported improvement using a range of related measures; standardized, pooled findings were translated onto a 6-point improvement scale. RESULTS: The review included 177 randomized controlled trials with 34,808 participants, including 26 trials of scalp psoriasis and 6 trials of inverse and/or facial psoriasis. Typical trial duration was 3 to 8 weeks. When compared with placebo (emollient base), the average improvement for vitamin-D analogues and potent corticosteroids was approximately 1 point, dithranol 1.2 points, very potent corticosteroids 1.8 points, and combined vitamin-D analogue plus steroid 1.4 points once daily and 2.2 points twice daily. However, these are indicative benefits drawn from heterogeneous trial findings. Corticosteroids were more effective than vitamin D for treating psoriasis of the scalp. For both body and scalp psoriasis, potent corticosteroids were less likely than vitamin D to cause skin irritation. LIMITATIONS: Reporting of benefits, adverse effects, and safety assessment methods was often inadequate. In many comparisons, heterogeneity made the size of treatment benefit uncertain. CONCLUSIONS: Corticosteroids are as effective as vitamin-D analogues and cause less skin irritation. However, further research is needed to inform long-term maintenance treatment and provide appropriate safety data.

Topical treatments for chronic plaque psoriasis.

BACKGROUND: Chronic plaque psoriasis is the most common type of psoriasis, and it is characterised by redness, thickness, and scaling. First-line management of chronic plaque psoriasis is with topical treatments, including vitamin D analogues, topical corticosteroids, tar-based preparations, dithranol, salicylic acid, and topical retinoids. OBJECTIVES: To compare the effectiveness, tolerability, and safety of topical treatments for chronic plaque psoriasis, relative to placebo, and to similarly compare vitamin D analogues (used alone or in combination) with other topical treatments. SEARCH METHODS: We updated our searches of the following databases to February 2011: the Cochrane Skin Group Specialised Register, CENTRAL in The Cochrane Library (2011, Issue 2), MEDLINE (from 1948), EMBASE (from 1980), Science Citation Index (from 2008), Conference Proceedings Citation Index - Science (from 2008), BIOSIS (from 1993), Dissertation Abstracts via DialogClassic (all publication years), and Inside Conferences (all publication years).We identified ongoing and unpublished studies from the UK Clinical Research Network Study Portfolio and the metaRegister of Controlled Trials. We checked the bibliographies of published studies and reviews for further references to relevant trials, and we contacted trialists and companies for information about newly published studies.A separate search for adverse effects was undertaken in February 2011 using MEDLINE and EMBASE (from 2005).Final update searches for both RCTs and adverse effects were undertaken in August 2012. Although it has not been possible to incorporate RCTs and adverse effects studies identified through these final searches within this review, we will incorporate these into the next update. SELECTION CRITERIA: Randomised trials comparing active topical treatments against placebo or against vitamin D analogues (used alone or in combination) in people with chronic plaque psoriasis. DATA COLLECTION AND ANALYSIS: One author extracted study data and assessed study quality. A second author checked these data. We routinely contacted trialists and companies for missing data. We also extracted data on withdrawals and on local and systemic adverse events. We defined long-term trials as those with a duration of at least 24 weeks. MAIN RESULTS: This update added 48 trials and provided evidence on 7 new active treatments. In total, the review included 177 randomised controlled trials, with 34,808 participants, including 26 trials of scalp psoriasis and 6 trials of inverse psoriasis, facial psoriasis, or both. The number of included studies counted by Review Manager (RevMan) is higher than these figures (190) because we entered each study reporting a placebo and an active comparison into the 'Characteristics of included studies' table as 2 studies.When used on the body, most vitamin D analogues were significantly more effective than placebo, with the standardised mean difference (SMD) ranging from -0.67 (95% CI -1.04 to -0.30; 1 study, 119 participants) for twice-daily becocalcidiol to SMD -1.66 (95% CI -2.66 to -0.67; 1 study, 11 participants) for once-daily paricalcitol. On a 6-point global improvement scale, these effects translate into 0.8 and 1.9 points, respectively. Most corticosteroids also performed better than placebo; potent corticosteroids (SMD -0.89; 95% CI -1.06 to -0.72; I² statistic = 65.1%; 14 studies, 2011 participants) had smaller benefits than very potent corticosteroids (SMD -1.56; 95% CI -1.87 to -1.26); I² statistic = 81.7%; 10 studies, 1264 participants). On a 6-point improvement scale, these benefits equate to 1.0 and 1.8 points, respectively. Dithranol, combined treatment with vitamin D/corticosteroid, and tazarotene all performed significantly better than placebo.Head-to-head comparisons of vitamin D for psoriasis of the body against potent or very potent corticosteroids had mixed findings. For both body and scalp psoriasis, combined treatment with vitamin D and corticosteroid performed significantly better than vitamin D alone or corticosteroid alone. Vitamin D generally performed better than coal tar, but findings relative to dithranol were mixed. When applied to psoriasis of the scalp, vitamin D was significantly less effective than both potent corticosteroids and very potent corticosteroids. Indirect evidence from placebo-controlled trials supported these findings.For both body and scalp psoriasis, potent corticosteroids were less likely than vitamin D to cause local adverse events, such as burning or irritation. Combined treatment with vitamin D/corticosteroid on either the body or the scalp was tolerated as well as potent corticosteroids, and significantly better than vitamin D alone. Only 25 trials assessed clinical cutaneous dermal atrophy; few cases were detected, but trials reported insufficient information to determine whether assessment methods were robust. Clinical measurements of dermal atrophy are insensitive and detect only the most severe cases. No comparison of topical agents found a significant difference in systemic adverse effects. AUTHORS' CONCLUSIONS: Corticosteroids perform at least as well as vitamin D analogues, and they are associated with a lower incidence of local adverse events. However, for people with chronic plaque psoriasis receiving long-term treatment with corticosteroids, there remains a lack of evidence about the risk of skin dermal atrophy. Further research is required to inform long-term maintenance treatment and provide appropriate safety data.

PROSPERO at one year: an evaluation of its utility.

BACKGROUND: PROSPERO, an international prospective register of systematic review protocols in health and social care, was launched in February 2011. After one year of operation we describe access and use, explore user experience and identify areas for future improvement. METHODS: We collated administrative data and web statistics and conducted an online survey of users' experiences. RESULTS: On 21 February 2012, there were 1,076 registered users and 359 registration records published on PROSPERO. The database usage statistics demonstrate the international interest in PROSPERO with high access around the clock and around the world. Based on 232 responses from PROSPERO users (response rate 22%), almost all respondents found joining and navigation was easy or very easy (99%); turn round time was good or excellent (96%); and supporting materials provided were helpful or very helpful (80%). The registration fields were found by 80% to be relevant to their review; 99% rated their overall experience of registering with PROSPERO as good or excellent. Most respondents (81%) had a written protocol before completing the registration form and 19% did not. The majority, 136 (79%), indicated they completed the registration form in 60 minutes or less. Of those who expressed an opinion, 167 (87%) considered the time taken to be about right. CONCLUSIONS: The first year of PROSPERO has shown that registration of systematic review protocols is feasible and not overly burdensome for those registering their reviews. The evaluation has demonstrated that, on the whole, survey respondents are satisfied and the system allows registration of protocol details in a straightforward and acceptable way. The findings have prompted some changes to improve user experience and identified some issues for future consideration.

The nuts and bolts of PROSPERO: an international prospective register of systematic reviews.

BACKGROUND: Following publication of the PRISMA statement, the UK Centre for Reviews and Dissemination (CRD) at the University of York in England began to develop an international prospective register of systematic reviews with health-related outcomes. The objectives were to reduce unplanned duplication of reviews and provide transparency in the review process, with the aim of minimizing reporting bias. METHODS: An international advisory group was formed and a consultation undertaken to establish the key items necessary for inclusion in the register and to gather views on various aspects of functionality. This article describes the development of the register, now called PROSPERO, and the process of registration. RESULTS: PROSPERO offers free registration and free public access to a unique prospective register of systematic reviews across all areas of health from all around the world. The dedicated web-based interface is electronically searchable and available to all prospective registrants. At the moment, inclusion in PROSPERO is restricted to systematic reviews of the effects of interventions and strategies to prevent, diagnose, treat, and monitor health conditions, for which there is a health-related outcome.Ideally, registration should take place before the researchers have started formal screening against inclusion criteria but reviews are eligible as long as they have not progressed beyond the point of completing data extraction.The required dataset captures the key attributes of review design as well as the administrative details necessary for registration.Submitted registration forms are checked against the scope for inclusion in PROSPERO and for clarity of content before being made publicly available on the register, rejected, or returned to the applicant for clarification.The public records include an audit trail of major changes to planned methods, details of when the review has been completed, and links to resulting publications when provided by the authors. CONCLUSIONS: There has been international support and an enthusiastic response to the principle of prospective registration of protocols for systematic reviews and to the development of PROSPERO.In October 2011, PROSPERO contained 200 records of systematic reviews being undertaken in 26 countries around the world on a diverse range of interventions.

Topical treatments for chronic plaque psoriasis.

BACKGROUND: Chronic plaque psoriasis is the most common type of psoriasis and is characterised by redness, thickness and scaling. First line management of chronic plaque psoriasis is with topical treatments, including vitamin D analogues, topical corticosteroids, tar-based preparations, dithranol, salicylic acid and topical retinoids. OBJECTIVES: To compare the effectiveness, tolerability and safety of topical treatments for chronic plaque psoriasis with placebo; to compare vitamin D analogues with other topical treatments. SEARCH STRATEGY: The Cochrane Skin Group's Trials Register was searched (2004/12). To update an unpublished 2002 review we also searched CENTRAL in The Cochrane Library (Issue 1,2005); MEDLINE (to 2005/02); EMBASE (to 2005/08); Science Citation Index (to 2005); Biosis (to 2005); Dissertation Abstracts (all publication years); Inside Conferences (all publication years); SIGLE (to 2005); National Research Register (all projects with a start date of 2001 to 2005); metaRegister of Current Controlled Trials. SELECTION CRITERIA: Randomised trials comparing treatments against placebo or against vitamin D analogues in people with chronic plaque psoriasis. DATA COLLECTION AND ANALYSIS: One author extracted study data and assessed study quality. A second author checked these data. We routinely contacted triallists and companies for missing data. We extracted data on withdrawals and adverse events. MAIN RESULTS: The review included 131 RCTs with 21,448 participants. Vitamin D was significantly more effective than placebo, although there was a wide variation in effect size with the standardised mean difference (SMD) ranging from -0.82 (95% CI -1.34 to -0.29) to -1.90 (95% CI -2.09 to -1.71). With one exception, all corticosteroids performed better than placebo, with potent corticosteroids (SMD: -0.95 (95% CI: -1.11 to -0.80; I(2): 61.1%; 17 studies; 2386 participants)) having smaller benefits than very potent corticosteroids (SMD: -1.29 (95% CI: -1.45 to -1.13; I(2): 53.2%; 11 studies; 1571 participants)). Dithranol and tazarotene performed better than placebo. Head-to-head comparisons of vitamin D against potent or very potent corticosteroids found no significant differences. However, combined treatment with vitamin D /corticosteroid performed significantly better than either vitamin D alone or corticosteroid alone. Vitamin D performed better than coal tar, but findings relative to dithranol were mixed. Potent corticosteroids were less likely than vitamin D to cause local adverse events. No comparison of topical agents found a significant difference in systemic adverse effects. AUTHORS' CONCLUSIONS: Corticosteroids perform as well as vitamin D analogues and are associated with a lower incidence of local adverse events. Further research is required to inform long-term maintenance treatment.